One field in which real time tracking of object motion can be advantageously employed, but is not currently greatly utilized, is in the field of sports. For example, continuous tracking of the motion of a tennis ball during a tennis match can provide a wealth of valuable information, including, for example, information about the skill or strategy of a player, because information such as ball speed and ball placement would be readily obtainable therefrom. Moreover, the trajectory of the ball, for example, obtained through such real time tracking, can be advantageously used to obtain other information of interest, and can also be used to form the basis for virtual camera views of play from any desired position, as well as forming the basis for virtual replays of portions of the match.
Historically, the only conventional form of tennis ball tracking that was available was the radar gun, whose operation is fully familiar to those skilled in the art and which could be used simply to measure the speed of the tennis ball, such as, for example, to determine the speed of service by a given player. But recently, in U.S. Pat. No. 6,233,007, “Method and Apparatus for Tracking Position of a Ball in Real Time,” issued on May 15, 2001 to I. Carlbom et al. (hereinafter, “Carlbom et al.”) a method and apparatus for tracking moving objects used in connection with athletic activities or sporting events, especially, balls, pucks, and the like, was disclosed. More particularly, the disclosure provides an apparatus and method for obtaining information corresponding to the object being tracked, which information may be advantageously used in a computer-generated replay of the sporting event (or some part thereof) using computer generated characters moving in accordance with motion data collected from real time tracking of the sporting event, and/or may be advantageously used to determine certain characteristics of the sporting event, such as ball speed, ball landing positions, etc. In particular, the disclosed method and apparatus is intended to be especially useful for enhancing television coverage of sporting events. U.S. Pat. No. 6,233,007, which is commonly assigned to the assignee of the present invention, is hereby incorporated by reference as if fully set forth herein.
Specifically, the method disclosed in U.S. Pat. No. 6,233,007 includes the steps of differencing present and previous frames of a video image including, for example, the ball, to obtain motion regions; converting the motion regions to a hue-saturation-intensity (HSV) color space; extracting the region corresponding to the ball based on empirical color data about the ball; obtaining a motion vector based on the motion of the ball region from a previous frame to the current frame; and updating the ball trajectory based on the newest motion vector obtained. The disclosed method also preferably includes a step of identifying completed trajectories based on preset constraints, and is preferably expanded on by using a at least one pair of cameras to provide a three-dimensional trajectory and sometimes preferably expanded on by using a plurality of cameras, especially a plurality of pairs of cameras. An apparatus according to the invention disclosed in U.S. Pat. No. 6,233,007 includes at least one camera connected to a computer which operates to difference previous and current frames, compute the ball track, convert ball regions to HSV color space and output the tracking and video information. And in a case where one or more pairs of cameras are used, the computer is preferably also provided with a stereo matching device or module for matching the tracking results from respective cameras and/or respective pairs of cameras.
Other (earlier) prior art had been generally directed to the tracking of people, which may be advantageously used in various applications such as, for example, in telecommunications applications, in store (e.g., supermarket) security applications, or in following the movement of players in tennis matches. The latter two applications are embodied, for example, in U.S. Pat. No. 5,764,283, “Method and Apparatus for Tracking Moving Objects in Real Time Using Contours of the Objects and Feature Paths,” issued on Jun. 9, 1998 to S. Pingali et al., where a method and apparatus for tracking moving objects, such as people, in real time is disclosed in which local features, such as extrema of curvature on boundary contours, are tracked, and trajectories of motion are derived by dynamically clustering the paths of motion of the local features. Specifically, existing feature paths from a current video frame are grouped together with preexisting clusters from previous video frames by selecting, as a candidate cluster to a specified feature path, the cluster having the closest relative distance to the specified feature path. U.S. Pat. No. 5,764,283, which is commonly assigned to the assignee of the present invention, is hereby incorporated by reference as if fully set forth herein.
And then, in U.S. Pat. No. 6,005,610, “Audio-Visual Object Localization and Tracking System and Method Therefor,” issued on Dec. 21, 1999 to S. Pingali (hereinafter, “Pingali”), a method for integrated audio-visual localizing and tracking of an object (e.g., a person) with particular application to telecommunications is disclosed. Specifically, the method includes the steps of capturing and transmitting an image of a video scene using a camera at an instant of time; identifying an object contained in the image having a preselected visual feature; and estimating a location of the object by determining an angular orientation relative to the image plane of the camera of an imaginary line extending from an optical center of the camera to a point on the image plane of the camera representing a portion of the object. The disclosed method further includes the steps of converting acoustic waves from an audio source into audio signals using at least two microphones at substantially the same time, and identifying the audio source by determining a locus of points representing an estimate of the location of the audio source on the basis of the audio signals. An improved estimate of the location of the object is then advantageously computed by determining the location of a region of intersection between the imaginary line and the locus. U.S. Pat. No. 6,005,610, which is commonly assigned to the assignee of the present invention, is hereby incorporated by reference as if fully set forth herein.
And lastly, FastCAM Replay, LLC, is a company which offers ultra-high-speed digital cameras for use in television sports coverage. In particular, FastCAM Replay digital cameras record up to 500 images-per-second—significantly more than the conventional 90 frames-per-second “Super Slow Mo” cameras previously used in sports coverage—to determine, for example, the landing spot of a tennis ball on or near court lines with great accuracy. Unfortunately, the very high speed camera approach used is not suitable for continuous tracking applications because these cameras provide a very narrow field of view and are very costly—thus, practical considerations (e.g., number of required cameras and the associated cost) restrict the use of such a technique to limited applications such as line calling.
Note that the above-described sports related prior art tracking systems which use video cameras assume that the object being tracked (e.g., a tennis ball) is visible from a small set of cameras positioned around a stadium. While the ball is, in fact, typically visible when in flight, it is often obscured when the opposing player approaches the ball with a racket to return the ball by a volley or ground stroke. Furthermore, since the video cameras sample the environment at discrete time intervals, the exact location where the ball hits the ground may not be recorded even when the ball is fully visible from the ball tracking cameras. Current methods typically calculate the point of ball impact by extrapolating the calculated ball trajectory to intersect the ground plane, leading to substantial inaccuracy in the determination of the exact ball landing position.
As a result of the above, current tennis ball tracking systems are essentially limited to tracking the ball of a serve, and cannot adequately continue tracking the ball without the determination of the exact time of when and/or position of where the ball changes direction whenever it, in fact, does so.